Searching for more products like a specified product

ABSTRACT

Using a processor, receiving, a query including a query image. A database is searched for a set of images similar to the query image, using a two-pass search. The results of the searching are then provided, the results including image members of the set of images similar to the query image. The first pass may be performed using a TF-IDF algorithm and the second pass ranks a predetermined number of results from the first pass by a best match algorithm or other type of algorithm. The type of second pass algorithm may be selectable by a user. If an end signal is not detected, a further two-pass search may be made based on a subsequent query having a subsequent query image. The subsequent query image may be one of the results of the previous two pass search. A subsequent query image may change from an image of a product of a first product category to an image of a product of a second product category.

RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication Ser. No. 61/318,766, filed on Mar. 29, 2010, which isincorporated herein by reference in its entirety.

BACKGROUND

As the use of network-based publication systems and marketplaces, suchas on-line commerce services or auction services expands, and the volumeof item listings in such applications increases, the speed, ease, andconvenience with which information can be retrieved from suchmarketplaces increases in importance to customers.

Item listings in such network-based marketplaces typically includedetails of a particular item which is, for example, up for sale orauction. These details are typically stored in text format and include adescription of the item together with other information, such as theprice, useful to a potential buyer in assessing the item listing. Inaddition, item listings often include visual material related to theitem, typically in the form of a photograph, drawings, or video clips.

The use of images in this context has customarily been limited to theprovision of information about item listings to customers, but it wouldbe useful to employ images associated with item listings for additionalpurposes, such as for image-based searching or for the automaticidentification of images for fraud protection purposes.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. Some embodiments are illustrated by way of exampleand not limitation in the figures of the accompanying drawings in which:

FIG. 1 is a block diagram illustrating a publication system in theexample form of network-based marketplace system according to an exampleembodiment.

FIG. 2 is a diagrammatic representation of marketplace and paymentapplications which may form part of the example embodiment of FIG. 1.

FIG. 3 is a diagrammatic representation of functional models anddatabases which may form part of the example embodiment of FIG. 1.

FIG. 4 is a flow chart illustrating an example method to index a listingimage according to an example embodiment,

FIG. 5 is a flow chart illustrating an example method to perform animage-based search in the example embodiment of FIG. 1.

FIG. 6 is a schematic view of operations which may be performed on animage for indexing thereof according to an example embodiment.

FIG. 7 is a schematic view of operations which may be performed on animage for indexing thereof according to another example embodiment.

FIG. 8 is a schematic of operations which may be performed on an imagefor indexing thereof according to yet another example embodiment.

FIG. 9 is a schematic of operations which may be performed on an imagefor indexing thereof according to yet a further example embodiment.

FIG. 10A is grid of pixels useful in illustrating a color method inpre-compiling digests.

FIG. 10B is an illustration of a method of using greyscale histogramsfor gradient detection.

FIGS. 11A to 11D is an illustration of segmentation and backgroundextraction from a color source image.

FIG. 12A is a top view of an HLS cone.

FIG. 12B is an isometric view of an HLS cone.

FIG. 13 is a sixteen-color grey matrix useful in texture detection.

FIG. 14A is an illustration of rotation normalization useful inpre-computing digests.

FIG. 14B is an illustration of translation normalization useful inpre-compiling digests.

FIG. 15 is an illustration of training sets useful in product categoryoptimization in selecting recall sets for a similarity search.

FIG. 16 is an illustration of a first ecommerce site page showingvarious entries for a similarity search.

FIG. 17 is an illustration of a second ecommerce site page showing partof a similarity search.

FIG. 18 is an illustration of a third ecommerce site page showing partof a similarity search.

FIG. 19 is a flowchart showing various operations in a similaritysearch.

FIG. 20 is a block diagram of a machine in the example form of acomputer system within which a set of instructions, for causing themachine to perform any one or more of the methodologies discussedherein, may be executed.

FIG. 21 is a diagrammatic view of a data structure according to anexample embodiment of a network-based marketplace.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of some example embodiments. It will be evident, however,to one skilled in the art that the present embodiments may be practicedwithout these specific details.

Index sets are associated with their respective images in a so-calledreverse indexing arrangement, in which an index database contains asingle entry for each unique index set or feature vector, with all theimages that contain that particular index set or feature vector beinglisted against the entry.

When, for instance, a query image is subsequently presented in order tosearch for identical or similar images in the image database, the queryimage is itself parsed or processed to generate multiple feature vectorsor sets of index values for the query image. The index sets for thequery image are thus generated in a similar manner to the generation ofindex sets for the index database.

To identify images similar to the query image, all of the query image'sindex sets are compared to index sets in the index database. Whencommonality is identified between a query image index set and a databaseindex vector, a hit count is incremented for each of the imagesassociated with that index set in the index database. The databaseimages are ranked in order of hit count, so that the result of thesearch are those item listings whose images have index sets showing thegreatest commonality with the index sets of the query image.

In an example embodiment, the index sets are generated by firstperforming an edge detection algorithm on the image and then normalizingthe image. Thereafter, the image is divided into a raster of cells orsub-portions at a resolution which is usually significantly lower thanits native pixel resolution, so that each cell or sub-portion comprisesmany image pixels. The raster may typically be a 10 by 10 grid of cells.Next, an index value is assigned to each cell based on an imageattribute, typically a light intensity value, of the underlying imagepixels. The result is therefore a grid of index values. Thereafter, setsof index values i.e. feature vectors) are created from respectiveoverlapping portions of the grid of index values. This process can beexplained as a sliding window that is moved over the grid to isolate twodimensional selections or portions of index values at a time, theportions overlapping and covering the entire grid. The index values ineach selection or portion together form one feature vector or set ofindex values. In an example embodiment, the moving window is three bythree cells in size, so that each feature vector is constituted by nineindex values.

The system and method thus provides for image comparison in apublication system, such as a network-based marketplace, which requiressignificantly less processing power, and is therefore faster than imagecomparison based on native image data, while returning results withrelatively high accuracy. The method is furthermore relatively robustand insensitive to scaling and resolution loss. A two-pass search isperformed, based on a query that includes a query image. The results ofthe searching are provided, the results including image members of theset of images similar to the query image. If an end signal is notreceived, a further two-pass search can be performed based on asubsequent query having a subsequent query image. The subsequent queryimage may be one of the set of images similar to the query image.

Architecture

One example embodiment of a distributed network implementing image-basedindexing for item listings in a network-based marketplace is illustratedin the network diagram of FIG. 1, which depicts a system 10 using aclient-server type architecture. A commerce platform, in the exampleform of a network-based marketplace platform 12, provides server-sidefunctionality, via a network 14 (e.g., the Internet) to one or moreclients. As illustrated, the platform 12 interacts with a web client 16executing on a client machine 20 and a programmatic client 18 executingon a client machine 22. In one embodiment, web client 16 is a webbrowser, but it may employ other types of web services.

Turning specifically to the network-based marketplace platform 12, anApplication Program Interface API) server 24 and a web server 26 arecoupled to, and provide programmatic and web interfaces respectively to,one or more application servers 28. The application servers 28 host oneor more marketplace applications 30 and payment applications 32. Theapplication servers 28 are, in turn, shown to be coupled to one or moredatabases servers 34 that facilitate access to a number of databases, inparticular an item listing database 35, an image database 36, and anindex database 37. The item listing database 35 stores data indicativeof item listings for items which are offered for sale or auction on theplatform 12. Each item listing includes, inter alia, a text descriptionof the relevant item and metadata categorizing the item. The imagedatabase 36 includes images associated with respective item listings inthe item listing database 35. The images in the image database 36 may bestandard format image files such as JPEG files. The index database 37contains index data relating to images in the image database to permitimage-based searching of the image database 36. The format of index datain the index database is described in more detail below.

The marketplace applications 30 provide a number of marketplacefunctions and services to users that access the marketplace platform 12.The payment applications 32 likewise provide a number of paymentservices and functions to users. The payment applications 32 may allowusers to quantify for, and accumulate, value (e.g., in a commercialcurrency, such as the U.S. dollar, or a proprietary currency, such as“points”) in accounts, and then later to redeem the accumulated valuefor products (e.g., goods or services) that are made available via themarketplace applications 30. While the marketplace and paymentapplications 30 and 32 are shown in FIG. 1 to both form part of thenetwork-based marketplace platform 12, it will be appreciated that, inalternative embodiments, the payment applications 32 may form part of apayment service that is separate and distinct from the marketplaceplatform 12.

Further, while the system 10 shown in FIG. 1 employs a client-serverarchitecture, the present invention is of course not limited to such anarchitecture, and could equally well find application in a distributed,or peer-to-peer, architecture system. The various marketplace andpayment applications 30 and 32 could also be implemented as standalonesoftware programs, which do not necessarily have networkingcapabilities. Additionally, while example embodiments are described withrespect to a marketplace platform 12, alternative embodiments may becontemplate use on a publication platform or other non-commerceplatforms,

The web client 16, it will be appreciated, accesses the variousmarketplace and payment applications 30 and 32 via the web interfacesupported by the web server 26. Similarly, the programmatic client 18accesses the various services and functions provided by the marketplaceand payment applications 30 and 32 via the programmatic interfaceprovided by the API server 24. The programmatic client 18 may, forexample, be a seller application (e.g., the TurboLister applicationdeveloped by eBay Inc., of San Jose, Calif.) to enable sellers to authorand manage listings on the marketplace platform 12 in an off-linemanner, and to perform batch-mode communications between theprogrammatic client 18 and the network-based marketplace platform 12.

FIG. 1 also illustrates a third party application 38, executing on athird party server machine 40, as having programmatic access to thenetwork-based marketplace via the programmatic interface provided by theAPI server 24. For example, the third party application 38 may,utilizing information retrieved from the network-based marketplaceplatform 12, support one or more features or functions on a websitehosted by the third party. The third party website may, for example,provide one or more promotional, marketplace or payment functions thatare supported by the relevant applications of the network-basedmarketplace platform 12.

FIG. 2 is a block diagram illustrating multiple marketplace and paymentapplications 30 and 32 that, in one example embodiment, are provided aspart of the network-based marketplace platform 12. The marketplaceplatform 12 may provide a number of listing and price-setting mechanismswhereby a seller may list goods or services for sale, a buyer canexpress interest in or indicate a desire to purchase such goods orservices, and a price can be set for a transaction pertaining to thegoods or services. To this end, the marketplace applications 30 areshown to include at least one publication application 41 and one or moreauction applications 44 which support auction-format listing and pricesetting mechanisms (e.g., English, Dutch, Vickrey, Chinese, Double,Reverse auctions etc.). The various auction applications 44 may alsoprovide a number of features in support of such auction-format listings,such as a reserve price feature whereby a seller may specify a reserveprice in connection with a listing and a proxy bidding feature whereby abidder may invoke automated proxy bidding.

A number of fixed-price applications 46 support fixed-price listingformats (e.g., the traditional classified advertisement-type listing ora catalogue listing) and buyout-type listings. Specifically, buyout-typelistings (e.g., including the Buy-It-Now BIN) technology developed byeBay Inc., of San Jose, Calif.) may be offered in conjunction with anauction-format listing, and allow a buyer to purchase goods or services,which are also being offered for sale via an auction, for a fixed-pricethat is typically higher than the starting price of the auction,

Store applications 48 allow sellers to group their listings within a“virtual” store, which may be branded and otherwise personalized by andfor the sellers. Such a virtual store may also offer promotions,incentives and features that are specific and personalized to a relevantseller.

Reputation applications 50 allow parties that transact utilizing thenetwork-based marketplace platform 12 to establish, build and maintainreputations, which may be made available and published to potentialtrading partners. Consider that where, for example, the network-basedmarketplace platform 12 supports person-to-person trading, users mayhave no history or other reference information whereby thetrustworthiness and credibility of potential trading partners may beassessed. The reputation applications 50 allows a user, for examplethrough feedback provided by other transaction partners, to establish areputation within the network-based marketplace platform 12 over time.Other potential trading partners may then reference such a reputationfor the purposes of assessing credibility and trustworthiness.

Personalization applications 52 allow users of the marketplace platform12 to personalize various aspects of their interactions with themarketplace platform 12. For example a user may, utilizing anappropriate personalization application 52, create a personalizedreference page at which information regarding transactions to which theuser is or has been) a party may be viewed. Further, a personalizationapplication 52 may enable a user to personalize listings and otheraspects of their interactions with the marketplace and other parties.

In one embodiment, the network-based marketplace platform 12 may supporta number of marketplaces that are customized, for example, for specificgeographic regions. A version of the marketplace may be customized forthe United Kingdom, whereas another version of the marketplace may becustomized for the United States. Each of these versions may operate asan independent marketplace, or may be customized or internationalized)presentations of a common underlying marketplace.

Navigation of the network based-marketplace may be facilitated by one ormore navigation applications 56. For example, a keyword searchapplication 57 enables keyword searches of listings published via themarketplace platform 12. Similarly, an image search application 59enables an image-based search of item listings published via themarketplace platform 12. To perform an image-based search, a user willtypically submit a query image, whereupon the image search application59 may compare the query image to images in the image database toproduce a result list of item listings based on a similarity rankingbetween the query image and the images associated with the respectiveitem listings. The comparison ranking is established by parsing orprocessing the query image to provide index data, and thereaftercomparing the query image's index data to pre-compiled index data forthe listing images, as described in more detail below. A browseapplication allows users to browse various category, catalogue, orinventory data structures according to which listings may be classifiedwithin the marketplace platform 12. Various other navigationapplications may be provided to supplement the search and browsingapplications.

In order to make listings, available via the network-based marketplace,as visually informative and attractive as possible, as well as to enableimage-based searching, the marketplace applications 30 may include oneor more imaging applications 58, which users may use to upload imagesfor inclusion within listings. Images thus uploaded are stored in theimage database 36, each image being associatively linked to at least oneitem listing in the item listing database 35. One of the imagingapplications 58 also operates to incorporate images within viewedlistings. The imaging applications 58 may also support one or morepromotional features, such as image galleries that are presented topotential buyers. For example, sellers may pay an additional fee to havean image included within a gallery of images for promoted items.

The marketplace platform 12 may also include an image indexingapplication 61 to parse or process images uploaded via the imageapplication 58, as well as to parse or process query images submittedvia the image search application 59. The result of processing images bythe image indexing application 61 is index data which is stored in theindex database 37. Particular processes for indexing images, as well asthe format of index data, are discussed in more detail below.

Listing creation applications 60 allow sellers conveniently to authorlistings pertaining to goods or services that they wish to transact viathe marketplace platform 12, and listing management applications 62allow sellers to manage such listings. Specifically, where a particularseller has authored and/or published a large number of listings, themanagement of such listings may present a challenge. The listingmanagement applications 62 provide a number of features (e.g.,auto-relisting, inventory level monitors, etc.) to assist the seller inmanaging such listings. One or more post-listing management applications64 also assists sellers with a number of activities that typically occurpost-listing. For example, upon completion of an auction facilitated byone or more auction applications 44, a seller may wish to leave feedbackregarding a particular buyer. To this end, a post-listing managementapplication 64 may provide an interface to one or more reputationapplications 50, so as to allow the seller conveniently to providefeedback regarding multiple buyers to the reputation applications 50.

Dispute resolution applications 66 provide mechanisms whereby disputesarising between transacting parties may be resolved. For example, thedispute resolution applications 66 may provide guided procedures wherebythe parties are guided through a number of steps in an attempt to settlea dispute. In the event that the dispute cannot be settled via theguided procedures, the dispute maybe escalated to a third party mediatoror arbitrator.

A number of fraud prevention applications 68 implement various frauddetection and prevention mechanisms to reduce the occurrence of fraudwithin the marketplace. One of the fraud prevention applications 68 mayinclude automatic image comparison, by use of index data produced by theimage indexing application 61 and stored in the index database 37. Suchimage comparison may be used by the fraud prevention application 68automatically to detect listing images similar to the query image, andto alert a fraud assessor to such image listings, so that the humanassessor can examine the identified item listing for assessing whetheror not the identified item listing is a fraudulent listing.

Messaging applications 70 are responsible for the generation anddelivery of messages to users of the network-based marketplace platform12, such messages for example advising users regarding the status oflistings at the marketplace (e.g., providing “outbid” notices to biddersduring an auction process or providing promotional and merchandisinginformation to users).

Merchandising applications 72 support various merchandising functionsthat are made available to sellers to enable sellers to increase salesvia the marketplace platform 12. The merchandising applications 72 alsooperate the various merchandising features that may be invoked bysellers, and may monitor and track the success of merchandisingstrategies employed by sellers.

FIG. 3 is a high-level entity-relationship diagram, illustrating therelationship between the databases 35 to 37 and several functionalmodules forming part of the applications 30 and 32. The system includesa receiving module 80, which may form part of the image searchapplication 59 (FIG. 2), for receiving a listing query which includes aquery image. Query images which may be submitted to the receiving module80 are typically electronic image files in a standard format, such asJPEG, GIF, TIFF, etc.

The receiving module 80 is operatively connected to a processing module86, which may form part of the image indexing application 61, forprocessing images to generate index data for the images. An examplemethod of processing images by the processing module 86 to generate theindex data is described in more detail below. An image listing module82, which may form part of the listing creation application 60 of FIG.2, is operatively connected to the processing module 86 to communicateto the processing module 86 images forming part of newly created itemlistings. The system further includes a database crawler 84 which servesto interrogate the listing image database 36 in order to identify imagesin the listing image database 36 which have not yet been processed bythe image indexing application 61, and for which there is thus noassociated index data in the index database 37. It will be appreciatedthat indexing of images in accordance with an example embodiment may incertain instances be implemented in a network-based marketplace havinglarge numbers of existing listing images for which there are, of course,no index data. To permit image-based searching of the listing imagedatabase 36 through the use of index data, the database crawler maycontinually locate and submit un-indexed legacy images in the listingimage database 36 and pass them to the processing module for imaging. Itshould be noted accordingly that although image database 36 isillustrated in the figures by a single element, the image database 36may be provided by any number of databases distributed through anetwork.

The processing module 86 is configured to parse or process imagessubmitted to it to generate index data in the form of a number of afeature vector or index sets 88 for each image. In an exampleembodiment, each index set 88 comprises a sequence of nine index values,as shown in FIG. 3. These index sets 88 are also referred to as featurevectors or image fingerprints, and the terms “index set”, “set of indexvalues”, “feature vector,” and “digest” are used interchangeably in thisdocument to refer to the results of indexing of images by the processingmodule 86.

The processing module 86 is in communication with the index database 37,to communicate index data generated by the processing module 86 to theindex database 37 for storage. In an example embodiment, index data isrelated in the index database 37 to images stored in the listing imagedatabase 36 in a reverse-indexing format. The data format in the indexdatabase 37 may be in a spatial data structure, such as atwo-dimensional k-d tree, to facilitate searching of the database 37. Asillustrated in FIG. 3, the index database has stored therein a singleentry for each unique feature vector 88, with each entry listing all ofthe database images which share that feature vector 88. For example withreference to FIG. 3, it will be understood that one of the featurevectors 88 produced by processing of an image with the filename IMG#23,stored in the image database 36, is [0, 0, 0, 2, 6, 12, 15, 13, 15], andthat this feature vector is also one of the feature vectors 88 producedby processing of IMG#3012. It should be appreciated that multiplefeature vectors 88 are generated by the processing of each image, andthat a particular image will thus be listed in the index databaseagainst each of the multiple feature vectors 88 thus produced. Forinstance, the example given in FIG. 3 shows that the results of indexingof IMG#3012 include feature vector [0, 0, 0, 2, 6, 12, 15, 13, 15] andfeature vector [4, 10, 17, 13, 9, 0, 0, 13, 15].

Each entry in the index database 37 is linked to at least one image inthe listing image database 36. Further, the images stored in the imagedatabase 36 are linked to associated item listings in the item listingdatabase 35. As discussed above, each item listing may compriseinformation about a particular item which is offered for sale or auctionon the marketplace platform 12, such as a description of the item, atleast one category of the item, a brand name of the item, etc. In theexample embodiment, entries in the respective databases 35 to 37 arelinked by use of image filenames as linking data. A person skilled inthe art will appreciate that, in other embodiments, any appropriate datastructure (e.g. relational databases or tables) may be used to linkimages in the image database 36 to, on the one hand, respective itemlistings and to, on the other hand, index data in the index database 37.

A comparison module 90 is in communication with both the processingmodule 86 and the index database 37, to compare index data for a queryimage with the index data in the index database 37 for finding databaseimages similar to the query image. The comparison module 90, in use,produces a ranking of database images according to similarity to thequery image. This ranking is achieved by comparing the feature vectors88 of the query image, which is generated by the processing module 86,with feature vector entries in the index database 37 in accordance witha comparison method which is described in more detail below.

The system 10 also includes a result module 92 for delivering to a userthe results of a search query which includes a query image. The resultmodule 92 is configured to return query search results as item listingsranked in order of the similarity of their respective images to thequery image, as determined by the comparison module 90. The searchresults may be delivered by the user in a format of the user's choosing,such as, for instance, via e-mail or in http format on a web browserapplication.

FIG. 4 is a flowchart showing a method 100 for processing an image in anetwork-based marketplace in accordance with an example embodiment. Themethod 100 starts with the creation of an item listing, at block 102, bya user via the listing creation application 60 (FIG. 2) in customaryfashion. Creation of the item listing may include submission of an imagerelated to the image, typically being an image of the item which isoffered for sale or auction on the marketplace. The submitted image isreceived by the receiving module 80 at block 104.

The image may be an electronic image in standard file format, such asJPEG, which comprises a raster of pixels. Each pixel may have hue,saturation, and intensity values, in conventional fashion. It will beappreciated that images which were submitted earlier and which arestored in the image database 36, but which have not been indexed, may beprovided to the processing module 86 by the database crawler 84 of FIG.3, so that operation 102 in FIG. 4 may instead comprise submitting anun-indexed image from the image database 36 for processing.

The submitted image is then processed, at block 122, to generate indexdata for enabling index searching of database images. Processing of theimage will be described with respect of an example image 200 illustratedin FIG. 6. First, an edge detection and normalisation operation isperformed, at block 106, on the image 200, to produce a normalised edgeimage 202 (FIG. 6). Edge detection processing is well known in the artand any suitable edge detection algorithm may be employed in operation106. Normalisation of the image includes desaturation, so that thenormalised edge image 202 is a grey scale image. Normalization mayfurther include contrast stretching or dynamic range expansion, toachieve consistency in intensity ranges for images processed by theprocessing module 86. Furthermore, normalization of the image mayinclude re-alignment of the image 200 by automated correction of theimage's orientation.

The normalised edge image 202 is then partitioned or divided, at block108, into ceils or sub-portions 207, to form a grid 204 (FIG. 6). In theexample embodiment, the resolution of the grid 204 is ten by ten, sothat the grid 204 comprises ten rows often sub-portions or cells 207each. Because the size of each block or cell 207 is considerably largerthan that of the image's pixels, each cell 207 comprises a plurality ofpixels. It will be appreciated that the resolution of the grid maydiffer in other embodiments,

Thereafter, an index value 208 is assigned, at block 110, to each cell207 based on an image attribute of the underlying image pixels of thecell 207. In this example, the image attribute is intensity, typicallymeasured on a scale of 0 to 255, or alternatively 0 to 100, where apixel having a zero intensity is white and a pixel having intensityvalue of 255 (or 100, as the case may be) is black. The index value 208assigned to each cell 207 may thus be the average light intensity valueof the pixels constituting each cell 207. The output of operation 110 istherefore a ten-by-ten grid 206 of index values 208 based on theintensity values of the respective cells 207 (FIG. 6). The grid 206 canbe viewed as a two-dimensional histogram of the base image 200. It is tobe appreciated that the particular index values 208 and the particularindex sets 88 illustrated in FIGS. 3 and 6, and in other examples inthis document, are chosen arbitrarily for illustrative purposes and donot accurately reflect the underlying intensity values of theillustrated images,

In other example embodiments, other image attributes can be used as wellas or instead of the intensity value. For instance, colour values of thecells 207 may be calculated and indexed together with or instead of theintensity index values. An index value may for instance assigned to eachcell 207 based on the average hue of the cell's pixels. Instead,separate grids may be produced for red, green, and blue colour spaces,and index values based on the intensity values of the respective coloursin the cells may be assigned to the cells.

At block 112, feature vectors or index sets 88 are compiled from thegrid 206. Compilation of the index sets 88 comprises iterativelyisolating portions 210 of the grid 206 and listing the index values 208in each portion 210 in sequence, to provide an index set 88. Compilationof the index sets 88 can thus be described as a sliding, overlappingmask or window 210 which is three-by-three index values in size, andwhich iteratively isolates all possible contiguous three-by-threeselections in the grid 206, to generate respective index sets 88. Eachindex set 88 thus comprises a sequence or vector of nine index values208. Although only two of these index sets 88 are shown in FIG. 6, itwill be appreciated that the results of index set compilation for aten-by-ten grid 206 will be 64 index sets.

The index sets 88 thus generated are incorporated in the index database37. As explained above with reference to FIG. 3, the index database 37comprises a single entry for each unique feature vector or index set 88,with all images which contain that index set 88 being listed in theentry. To this end, each index set 88 generated in operation 112 isprocessed by first establishing, at block 114, whether or not the indexdatabase 37 already includes an entry for that particular index set 88.If the determination at block 114 is in the affirmative, the image whichis the subject of current processing is linked, at block 118, to theexisting entry in the index database 37 by including the image filenamein the listing of images in the respective database entry. If, however,the determination is in the negative and there is not yet a databaseentry for the index set 88 under consideration, a new index set entry iscreated, at block 116, in the index database 37. It will be appreciatedthat such a new database entry will have only the current imageassociated with the particular index set 88. This database writingsequence is looped, at block 120, through all of the index sets 88generated at operation 112, therefore being performed 64 times in theexample embodiment having a ten-by-ten grid 206.

FIG. 5 is flowchart of a method 130 of image-based searching inaccordance with an example embodiment. The method 130 is initiated byuser submission of a query, at block 132, via the image searchapplication 59 of FIG. 2, and includes a query image on which the searchis to be based. The query image is again a digital image in a standardfile format. A user wishing, for instance, to search for item listingsin respect of a particular device may photograph the target device, forexample by use of a mobile phone with image capturing capabilities, andmay submit the image to the image search application instead of or inaddition to entering text in the keyword search application 57. A searchfor listing images that are in database 37 may be made, based on thequery image, and a set of images similar to the query image is returned.

The query image is processed, at block 122, by the processing module 86to generate index sets 88 for the query image in a manner identical toindexing of images in accordance with the method 100 of FIG. 5. Edgedetection and normalisation of the image is thus performed, at block134, whereafter the query image is partitioned in a grid of cells 207,at block 136. Then, index values 208 are assigned to each cell 207 basedon the intensity values of the underlying pixels of the cell 207, andindex sets 88 are compiled by use of the sliding, overlapping window 210method described above.

After generation of the query index sets 88 from the query image, thequery feature vectors or index sets 88 are compared to the index data inthe index database 37 to indentify images similar or identical to thequery image. To this end, the comparison module 90 processes each of thequery index sets 88 in turn. The comparison module 90 steps or loops, atblock 146, through the index values 208 of a particular index set 88 tofind, at block 142, all index set entries in the index database 37 thatshare that index value 208. For each index entry identified as sharingthe particular index value 208 under consideration, a hit count isincremented for each of the images associated in the index database 37with the identified database entry. The database images are then rankedin descending order according to hit count.

In an example where the first query index set 88 is [24, 12, 13, . . . ,4], the first index value 208 is 24. If the index database 37 includesthe following entries:

1. [0, 0, 24, 16, 26, . . . ]=IMG#221, IMG#3224, IMG#6739

2. [36, 48, 18, . . . ]=IMG#644, IMG#2542

3. [24, 12, 0, 0, . . . ]=IMG#3224, IMG#2143,

the first iteration of operation 142, in respect of value 24, willresult in an increment in the hit count of the images in data entries 1and 3 above. The second iteration of operation 142, which will be inrespect of value 12, will result in the incrementing, at block 144, ofthe images in data entry 3 above. The results of looping through all theindex values 208 of query index set 88 shown above will be IMG#3224=3hits; IMG#2143=2 hits, with the remainder of the listed imagesregistering a single hit, apart from IMG#644 and IMG#2542 which willhave no hits registered against them.

After repeating operations 142 to 146 for all of the index values 208 ofone of the index sets 88, the process loops, at block 148, to the nextquery index set 88. Operations 142 to 148 are thus repeated until all ofthe index values of all of the query index sets 88 have been compared tothe index database entries, the hit counts being aggregated to provide aranking of images by hit count, at block 150.

In other embodiments, the comparison of index sets may include comparingall of the index values of query index sets with all of the values inthe respective pre-compiled index sets forming database entries in theindex database 37, to that a hit is registered only if there is completeoverlap between the index values of, on the one hand, the query indexset, and, on the other hand, the index values of the particular databaseentry. In yet further embodiments, the comparison of feature vectors orindex sets may include matching not only the values of the query indexsets to database entry index sets, but also matching the sequence ofindex values in the respective index sets. A hit will thus be registeredonly if the query index set matches a database entry's feature vector orindex set 88 exactly, in other words if both the values and the sequenceof the respective index sets are identical. To promote processing speedand efficiency when performing exact feature vector or index setmatching, the index value range may be reduced in scale, so that theindex values, for example, range in value from 0 to 10 instead of, forexample, from 0 to 100.

Instead, or in addition, the comparison operation may include aweighting of the hit count based on the position of the respective indexsets in the image. In other words, hits may be assigned weights based onadjacency of the index sets in the image. Two matching index sets whichwere compiled from image portions or windows which are in adjacent oridentical locations in the grid may therefore result in a higherweighted hit, while a lower weighted hit may be registered if therespective image portions or windows are less adjacent.

As mentioned above, pre-complied feature vectors or index sets 88 may bestored in the index database in a data structure like a k-dimensionaltree, also known as a k-d tree. Comparison of a query index set orfeature vector may in such cases comprise performing a nearest neighboursearch in the k-dimensional tree.

The hit counts of the images are passed by the comparison module 90 tothe result module 92. The result module 92 then displays to the user, atblock 152, the results of the search. The search results are provided asa list of item listings extracted from the item listing database 35, thedisplayed item listing being the item listings linked to the top rankedimages, as identified by the comparison module 90.

The system 10 and methods 100, 130 described above provides foreffective image-based searching in the network-based marketplace.Indexing of the images in the image database 36 in accordance with thedescribed example embodiment permits similarity comparison of the queryimage with large numbers of database images without requiringprohibitive processing power or time.

In addition to use of the indexing method 100 for user-initiatedimage-based searching of the database 36, it may, in other embodiments,be used for fraud prevention applications in the network-basedmarketplace. In such embodiments, the fraud prevention application 68,shown in FIG. 2, may be provided with a query image representing anarticle which may be susceptible to fraud. Image comparison as describedabove with reference to FIG. 5 may then automatically be performed inresponse to the creation of new listings, so as to flag new itemlistings having images with a similarity rating or weighting, asdetermined by index data comparison, higher than a set threshold value.

In other embodiments, the processing operation 122 to generate indexsets 88 may differ in a number of aspects, some of which are describedwith reference to FIGS. 7-9. In one embodiment, illustrated in FIG. 7,the indexing method 100 includes producing a number of variations of asubject image 200, and processing each of the variations to produce aplurality of index sets 88 for each of the variations. In other words, asingle base image 200 is used to produce multiple image variations, andeach of the image variations is indexed and its index data is recordedin the index database as separate images, each of which is linked to acommon item listing. Identification of any one of these image variationsranking images by hit count, at block 150 in FIG. 5, will result in theassociated item listing being presented in results of an image-basedquery.

In another embodiment, such image variation may be performed uponsearching instead of, or in addition to, image variation duringindexing. In such case, a query image may thus be processed to producemultiple image variations, index sets 88 thereafter being generated foreach of the variations and being compared to the index database 37.

In the embodiment illustrated in FIG. 7, the subject image is cropped atthree different magnification levels to produce three edge imagevariations 220 to 224. Each of these edge image variations 220 to 224are then partitioned into sub-portions 207 to provide respective grids226 to 230. Although not illustrated explicitly in FIG. 7, the grids 226to 230 are then assigned index values 208, and index sets 88 arecompiled as described with reference to FIGS. 4 and 6. It will beappreciated that the number of magnification levels, and therefore thenumber of image variations 220 to 224 can be varied. In one embodiment,which is not illustrated, fen image variations based on varying croppingmagnifications may be produced.

In another embodiment, illustrated in FIG. 9, the subject image 200 isrotated or angularly displaced at three different angles to producethree edge image variations 250 to 254. These edge image variations 250to 254 are then partitioned into grids 260 to 264 for further processingto generate respective collections of index sets 88.

In yet another embodiment, illustrated in FIG. 8, the subject image isfirst subjected to edge detection and normalisation, to produce anormalised edge image 202. Thereafter, the normalised edge image 202 ispartitioned at three different grid resolutions. In the exampleembodiment, the image 202 is partitioned at a 5×5 resolution to producea first grid 240; it is partitioned at a 10×10 resolution to produce asecond grid 242; and it is partitioned at a 20×20 resolution to producea third grid 244. Each of these grids is further processed to producerespective collections of index sets 88, and each of the variations240-244 may be recorded in the index database 37. It will be appreciatedthat each of the variations is linked to the common image listing, sothat identification of any of the variations 240 to 244 in animage-based search will result in return of an item listing associatedwith the subject image 200.

In another embodiment, the partitioning resolution for indexing may bedetermined by a category of the relevant item listing. For example, themethod may include categorising an item listing upon creation,determining the item listing's category before processing the image, atblock 108, and selecting the partitioning resolution based on the itemlisting category. For instance, apparel may be partitioned at a 10×10resolution, while electronic devices may be partitioned at a 15×15resolution.

It will further be appreciated that the system and methodology describedabove can be applied to video content as well as or instead of imagedata. The method may in such case include extracting images from videofiles, and processing the extracted images in accordance with theexample embodiments described herein. Image extraction from such videocontent may include automatically identifying scene changes in the videocontent, (e.g. by comparison of successive frames or images in the videocontent, and extracting images or frames at the start of such scenechanges.

The example embodiments described herein address some of the technicalchallenges associated with effective processing of images linked to itemlistings. For instance, image databases of network-based marketplacesare often very large, comprising millions of images, so that the timeand/or processing power consumed by conducting a search or imagecomparison in the database is prohibitive. However, a comparison ofindex data generated for the query image with index data of databaseimages is considerably less resource intensive.

In another example embodiment, there is provided a system and a methodto index images associated with item listings in a network-basedmarketplace, so that subsequent similarity search or comparisonoperations are performed on index data instead of on base images storedin an image database. Images in the image database are thus indexed byparsing or processing the images for feature extraction. The featureextraction may include generating multiple sets of index valuesassociated with each image. Each set of index values is also referred toas a feature vector or an image fingerprint.

Digests for various image features can be pre-computed and stored in theimage database for subsequent recall, as a recall set, for comparison toa query for an item, for example, at run time. As examples, a digest caninclude edge information as alluded to above, color information,pattern, quality, texture, and the like. Digests can also includetextual information as well as image information such as item attributesthat are either determined by the system at the time of listing, or arespecified by the seller. However, here “textual” does not mean merelywords typed by the user. “Textual” means text information to be storedabout the item the seller is listing, including item title, and alsoattributes of the item. This can be considered “structured” and“unstructured” item data. For example, unstructured data may includeitem title and item description, which may comprise whatever data theseller wants to include. Structured data may be data such as the itemcategory e.g., shoes, handbags, and the like) and item aspects such asbrand, material, and the like, where each piece of information can bestored as separate name-value pairs. For example, if the listing item isa Coach handbag with title “New brown Coach purse”, the digest mayinclude the following terms as textual elements of a pre-computed digest(new, brown, Coach, purse, Brand: Coach, Material: leather, Style:satchel).

As an example of an edge part of a digest, the image can be convertedinto a grid as discussed in detail below, where each box in the gridcontains a value indicating the presence of a signal, such as a pixel,which, when integrated across all grids of interest, indicate a strongedge in that region of the image. The edge digest is stored as a stringof buckets representing each element in a matrix. This is discussed inmore detail below,

Color Detection

In performing a color process, it is desirable in pre-computing digests,or for building a recall set for comparison with a query, to determinewhether a first pixel in a number of pixels bears a positive likeness toa second pixel or a negative likeness to the second pixel. This can beseen in FIG. 10A which depicts a grid of nine pixels, similar to portion210 on FIG. 6. A central pixel 124 can be used as a reference pixel.Neighbouring pixels are evaluated for gradient detection of theneighbouring pixels. If the pixels of FIG. 10A exhibit the gradientvalues indicated, it is seen that the values are changing rapidlybetween 100, 100, 100, and 150, 150, 150, so, for this example, avertical line can be placed as indicated in FIG. 10A, illustrating basicgradient detection. This grid computation can be done by processingmodule 86 for each pixel in the image, the type of line referenced abovebeing placed at or near rapid change of values for each grid.

A more refined process of gradient detection for an image is illustratedin FIGS. 10B and 11. Referring to FIG. 11 in more detail, segmentationand background extraction is performed on an image such as image 111.Image 111, also referred to as a source image, may be an image, such asa photograph, of an item that is to be offered for sale on anetwork-based marketplace. A digest can be pre-computed for the image111, and used for subsequent similarity searching, as alluded to above.In order to find the various regions of change of signal, such as apixel, of the image 111, refined gradient detection can be employed, asseen in FIG. 10B, using, for example, a circle 101 that is bisectedalong eight different diameter lines giving what can be viewed of eightwindows of computation. Each of the windows divides the circle 101 inhalf, creating what is shown here as Region 1 and Region 2, along linesof different angles (each 22.5 degrees apart, in one example). Thedistance between the greyscale histograms of the pixels that lie beneaththe two regions (the two halves of the circle 101, Region 1 and Region2) is computed using an algorithm. Stated another way, the difference inintensity of blackness versus whiteness of Region 1 versus Region 2 canbe computed and a number is assigned to the result of that computation.The diameter of the circle 101 can then be rotated, eight rotations of22.5 degrees, and the distance between the greyscale histogram of thepixels that lie between the two halves of the circle 101, Region 1 andRegion 2, of FIG. 10B is again computed. The diameter of the circle 101can be rotated another 22.5 degrees and the distance recomputed. This isrepeated until the diameter has been rotated 180 degrees and the maximumdistance has been computed at each point.

Returning to FIG. 11, the maximum difference of all of the above eightangles of the diameter can be computed for each pixel and that value canbe used in a resulting matrix for the image 111. The areas of blacknessare where there is substantially no change, whereas at edges of an imagethere is a major gradient difference. This matrix is sometimes calledthe gradient matrix. While an example using specific diameter rotations(e.g., 22.5 degrees and 180 degrees of rotation) has been given withrespect to FIG. 10B, it is recognized that other angular rotations, andtotal degrees of rotation, can be used to obtain the same or similarresult.

Edges of the image 111 can be found by a technique called watershedding. In a recomposing process, each area of an image is filled withcolor. This can be visualized generally as if the image were a terrainmap and as if the color were water. The points of a watershed image 113of FIG. 11B where the colors begin to meet create boundaries todifferentiate object from background. This is sometimes called“computing the watershed” from the gradient matrix. An analogy forcomputing the watershed is to picture the above gradient matrix as athree-dimensional landscape. The watershed analogy begins by floodingthe landscape starting at a zero gradient level and working up to themaximum gradient level. At each level, some segments, or regions (ofwater, in the analogy), will grow and new segments, or regions (ofwater, in the analogy) will form. Each of these segments will remain aseparate segment. Where two separate segments join, a line can be drawn.These lines comprise the watershed image 113 that is seen in FIG. 11B.This can be done in each segment of the watershed image 113.

Next, a segment graph 115 is created from the watershed image 113 asshown in FIG. 11C. Small segments are joined to a most-similar neighbor.Very similar adjacent segments are also joined. Then, border segmentsare marked as part of the background unless an average pixel is in themiddle third or most of the segment's pixels are in the middle third.The background is zeroed out from the original image, and a huehistogram is calculated to generate a compressed color representation117 for the image as shown in FIG. 11D.

As discussed previously, the color representation 117 of the image maybe accomplished with hue level color bucketing using an HLS cone. An HLScone is illustrated in FIG. 12A and 12B. FIG. 12A depicts a top view ofthe HLS cone, and FIG. 12B depicts an isometric view of the cone. Hue Hcan be viewed as being represented on a circumference 123 of a huecircle 121 at a depth, and in the visual spectrum red, orange, yellow,green, blue, indigo, and violet. On vertical axis V of FIG. 12B,luminosity is increasing upwardly with distance, The saturation Sincreases outwardly with the radius. In example embodiments, hue levelbucketing is based on the following heuristics:

-   -   A bottom region 125 of the cone is black.    -   At the top level of the cone, a center region 127 of the circle        121 is white.    -   A middle region 129 of the cone is grey.    -   On the rest of the image, the outer edge of the circumference        123 is divided into sixteen colour buckets.

For each image, such as the image 111 of FIG. 11, there are buckets foreach count of pixels in regions for (black), (grey), (white), and 16buckets corresponding to (hue) (luminosity) (saturation) as discussedabove. That is, for the color part of a digest, pixels of various huesof a color can be mapped into a single, discrete color bucket based ontheir hue. For example, various gradations of red pixels may be mappedto the “red” hue bucket, and so on for pixels of the various colors.This may be a color histogram which can be, as in one example, a stringof fifty-one (51) bytes that has the form: (black), (white), (grey),(hue1), (lightness1), (saturation 1), (hue2), (lightness2), (saturation2), . . . (hue 16), (lightness 16), (saturation16).

If the histograms are looked upon as vectors, the standard Euclidiandistance between the two vectors could be calculated. Unfortunately,this may yield an unreal effect because Euclidian distance indicatesthat each color bucket is weighed the same. That is, red is just assimilar to orange in Euclidian distance as red is to purple, or as redis to green, or as red is to black. But human perception does notoperate in that manner; color lies on a spectrum (red, orange, yellow,green, blue, indigo, and violet) in human vision such that red is closerto orange and yellow than red is to green, for example. Therefore, thecolor buckets may be weighted by the distance from each other as well,As such, the color buckets can be organized linearly around thecircumference 123 of hue circle 121, as seen in FIG. 12A, The colorbuckets are in order such that red, orange, yellow, and green match thecolor wheel. Therefore, if a first color bucket is next to a secondcolor bucket, then the first color has some visual-perceptual similarityto the second color.

The color similarity assumes a fixed distance between black, grey, andwhite (four units) and a distance between each color bucket is a unitdistance in the circular scale at the circumference 123. Stated anotherway, the circular hue space shown in FIGS. 12A and 12B is dividedequally and discretely into 16 regions, or “color buckets.” Three morebuckets can be added representing white, grey, and black. For each ofthese 19 buckets, three (3) values are computed: the count of pixelsfalling in the bucket, the average luminosity of those pixels, and theaverage saturation of those pixels. The distance between any two colorbuckets includes the number of “steps” separating them around thepartitioned hue circle 121. Thus, for example, if the “red” bucket isadjacent to the “orange” bucket, the distance between them is one unit.For black, grey, and white buckets, the distance to any other bucket maybe fixed at four units. Taking into account the distance between colorbuckets resolves the above human perception problems encountered whenusing only Euclidian distance.

This provides an improvement when two histograms are compared todetermine how far these histograms are from each other. That is, itenables the system to determine how much visual difference obtainsbetween two different histograms. In the comparison of histogramvectors, the color buckets are weighted by the distance between them byproviding, in the similarity comparison, that if a given color bucket isnext to another given color bucket, as in FIGS. 12A and 12B, the twocolors represented by the color buckets have similarity to each other.

Texture Detection

Texture detection can be performed by the system using grey levelco-occurrence analysis. The system can convert an image to a grey scale,for example, a 16-color grey matrix, numbered one through sixteen oneach coordinate, as illustrated in FIG. 13, The system can detect eachpixel of the image, examine neighbouring pixels surrounding each pixel,and count the occurrence of two pixels coming together, As an example,and with continued reference to FIG. 13, box 131 represents one pixel,grey value one, occurring next to another pixel having grey value one,The system counts how many times in an image grey value one (e.g., alight grey) occurs next to the exact same color grey (e.g., grey valueone). The same process may be performed for grey value two at box 133,grey value three at box 135, grey value four at box 137, . . . , greyvalue 16 at box 139. The matrix is filled out to get the colorco-occurrence of the greys. Then, the matrix is transformed. Thisprocess can be implemented in the pre-computing of digests, and also maybe done on the query at the time of the query request so that the queryrequest can be compared to the digests in the image digests in the indexdatabase 37. However, a pixel of one color is likely to occur next to apixel of the same color, because colors tend to come in blocks of shade.That being the case, the matrix tends to include data that is bunched asat box 139 of FIG. 13. Without additional scaling, this is notsignificantly useful data because the matrix provides information thatis primarily that the grey is co-occurrent, which is already knowninformation, Without further adapting, the matrices do not allowmeaningful differentiation of one image from another. The result wouldbe that each pixel seems very similar to others. However, the system canscale up the matrices (e.g., one in pre-computing digests and one incalculation on the query image) so that the system scales up the valuesin the corners of the matrices (e.g., one of the matrices illustrated inFIG. 13), corner values being the key values for determining texture.Stated another way, transformation emphasizes the areas of an image thatchange rather than the areas of an image that remain the same, whichemphasis makes the texture in an image stand out. The edges of an imagetend to contain the most relevant texture data,

The above matrix for the pre-computed digest and also for the queryimage at run time) can be transformed to emphasize high contrast data,by the following example equation:n _(x,y) =m _(x,y)((|x−y|)+1)²Where: m_(x,y) is the value in the original matrix at position (x,y);and

-   -   x and y represent grey values.        So, referring again to FIG. 13, at box 131, the coordinates of        which are one, one, nothing happens to box 131 because one minus        one is zero, and so on.        Rotation Normalization

As mentioned previously, when comparing two images, such as a queryinput from the user interface and digests from the index database 37,the two images are compared by taking edge information, placing oneimage over the other, and determining the differences in the images, asdiscussed above. However, the process may not be robust for rotation.For example, and with reference to FIG. 14A, the two images 141, 143 areseen to be at two different orientations. This may be due to the factthat the user taking a photograph of the item depicted in image 143 forthe query images is under no restriction as to what orientation of theitem to use in the photograph. If the system were to then place theimages 141 and 143 one over the other, the difference would be too greatfor meaningful computation for similarity and difference determination.That being the case, the system may rotate at least one of the images141 and 143 so that both images 141 and 143 have the same angularalignment. The computation including placing one image over the other todetermine similarity and differences can then proceed.

This rotation can be accomplished by using principal component analysis(PCA). PCA is a mathematical procedure that uses an orthogonaltransformation to convert a set of observations of possibly correlatedvariables into a set of values of uncorrelated variables calledprincipal components. The number of principal components is less than orequal to the number of original variables. This transformation isdefined in such a way that the first principal component has as high avariance as possible (e.g., accounts for as much of the variability inthe data as possible), and each succeeding component in turn has thehighest variance possible under the constraint that it be orthogonal to(e.g., uncorrelated with) the preceding components. Principal componentsare guaranteed to be independent only if the data set is jointlynormally distributed. PCA is sensitive to the relative scaling of theoriginal variables. However, in this instance, PCA is used to determinean angle or a direction in which the images are each spread out the mostso that similar objects have similar alignments. This method can beperformed by finding the two major Eigen vectors of each edge detectedof images 145 and 147, one the listing at digest creation time, and theother the query image at runtime. Then the system aligns one of themajor Eigen vectors, for example the first major Eigen vector, K1 inFIG. 14A, to the x-axis. The system can do this for each of two images(e.g., a listing image 148 at digest creation time and a query image 149at runtime), so that each image 148 and 149 then becomes aligned in thesame direction for comparison.

Translation Normalization

In addition, image features for shape may not be translation invariant.For example, a similarity measure for an image 140 in FIG. 14B, and asimilar sub-image 142 which is simply shifted from image 140, would bevery low. To solve this matter, the system takes the gradient of theedge detected image as previously discussed, and constructs a box,sometimes referred to as a “bounding box,” around the foreground object.A bounding box for a point set in N dimensions is the box with thesmallest area (or volume or hypervolume in higher dimensions) withinwhich all the points lay. The bounding box can be on a fixed threshold,although a dynamic threshold could be used as well. The system can chalkup the bounding box and then operate on sub-images. That is, the systemmay break an image into parts and treat those parts much like words in adocument. The system can detect edge images of the two images 144, 146and then divide the images 144, 1467 into a 12 by 12 cell shownsymbolically at 148, 150. So each image may be a number of 3×3 subsetsas discussed above with respect to the portion 210 of FIG. 6, and can beviewed as a nine-dimensional vector.

At digest create time, the system can operate on a large corpus oflisting images linked to sub-images to generate vectors as described,providing a large pool of sub-images, perhaps in the millions. Thesystem then runs a k-means clustering algorithm on the corpus ofsub-images. K-means clustering is a method of cluster analysis, Ak-means cluster algorithm assigns each point to the cluster whose center(also called a centroid) is nearest. The center is the average of allthe points in the cluster—that is, its coordinates are the arithmeticmean for each dimension separately over all the points in the cluster.

Example: The data set has three dimensions and the cluster has twopoints:

X=(x1,x2,x3) and Y=(y1,y2,y3),

-   -   Then the centroid Z becomes Z=(z1,z2,z3),    -   Where

${z_{1} = \frac{x_{1} + y_{1}}{2}},{z_{2} = {{\frac{x_{2} + y_{2}}{2}\mspace{14mu}{and}\mspace{14mu} z_{3}} = {\frac{x_{3} + y_{3}}{2}.}}}$

Applying a k-means cluster algorithm, to, for example, millions ofnine-dimensional vectors during digest creation will reduce the vectorsto a reasonable number of centroid points, perhaps as few as onethousand to ten thousand. These centroid points can be viewed as codepoints in a code book for each of the respective vectors. At runtime,when a user submits an image query to the system, the system looks ateach sub-image (e.g., sub-image 142 in FIG. 14B) and looks up theclosest codebook vector for each sub-image. This results in breaking animage into sub-images, performing a cluster audit, determiningcentroids, and using the centroids as a code book. Each of the centroidscan be viewed as a nine-character sequence or token. Consequently, acomplex image has been broken down into a set of nine-character vectorsto which standard search engine operations can now be easily applied forthe image query to return a recall set. This dramatically reducescompute time for recall. In the search for similarity to the imagequery, the system may request the search engine to return the imagesthat, out of millions of images, have the most tokens in common withthose of the sub-images of the image query.

Query Image Comparison at Run Time using a multi-pass similarity search

As discussed above, a listing query (also referred to as “query image”)can be subsequently presented by web client 16 of FIG. 1 in order tosearch for images in the index database 37 that are identical or similarto the listing query. The query image, referred to herein as a pivot, isitself parsed or processed to generate multiple feature vectors or setsof index values for the query image. The index sets for the query imageare thus generated in a similar manner to the way generation of indexsets, or digests, are pre-computed for the index database. The digest ofthe query image may be compared against the digests in the indexdatabase to determine digests similar to the query image digest.

Textual information which could include, for example, title andattributes, is stored with an item and can be used in the first-passquery based on textual similarity. For example, if the textualinformation of a pivot item included Title=“New brown Coach purse” andaspects Brand: Coach, Material: leather and Style=satchel, all thesevalues would be passed to a RANK operator as the “text query”. The RANKoperator compares the text query against images stored as pre-computeddigests in index database 37 using a IF-IDF like algorithm, so each termis weighted according to inverse document frequency, meaning terms thatare rare have higher weight. For example, if one in two items in thehandbags category contained the word “new”, then “new” would have an IDFscore of 2, whereas if Brand=Coach appears only 1 in 20 times in thehandbags category, then this term would have a weight of 20. The RANKoperator looks for items that match any of these terms, and gives ascore according to number of matches and associated term weights. Thenthe top N items from this step can be re-sorted using a second pass sortusing a second ranking method, such as a “Best Match” method like thosedisclosed in copending application Ser. No. 12/476,046, entitled“Methods and Systems for Deriving a Score with which hem Listings areOrdered” which is incorporated herein by reference in its entirety. Anexample of a second pass ranking that re-sorts these top N items mightuse a combination of image comparison operations (for example: distancebetween color histograms, discussed above, cosine similarity of edgematrices, pattern comparison, and the like), textual similarity, by time(sale ending soonest); by newly listed; by price+shipping cost (lowestfirst); by price+shipping cost (highest first); by best match; by price(highest first); by price (lowest first), and seller or item qualitymeasurements to produce a final ordering which is presented to users. Inthis case, as an example, the profiles for image similarity search canbe:

-   w_1*ColorSim(colordigest)+w_2*EdgeSim(edgedigest)+w_3*SimilarityScore    where

w is a weight customizable by pivot item category,

ColorSim is the distance between color digests of the two images,

EdgeSim is the distance between edge digests of the two images, and

SimilarityScore is the score computed above in the first-pass sort.

As discussed previously, both color and edge digests are represented ashistograms measuring the number of pixels falling into distinct bins:for color the bins are regions of a given color space, and for edge thebins are gradients of a certain magnitude. There are many ways offinding distances between two histograms. One way is to use histogramintersection for color, and cosine similarity for edge.Product Category Optimization in Similarity Searching

The weights “w” on each factor are optimized using feedback from usersas a way of optimizing sorts by item type or category. Datasets can beused comprising idealized orderings as ranked by human judges, and/or byclick-through patterns from an ecommerce website to optimize theimportance of each factor to users by product category. Then when a userselects an item for a query, the optimum weights for that item'scategory can be used to rank the results to be presented to the user.

The method of obtaining weights w, above, by human judges is discussedwith respect to FIG. 15. In general, the method is employed to determinethe importance of each attribute of an item, sometimes referred to as afactor, to a user interested in purchasing the item. This importance mayvary considerably depending on product type. Training sets can be sentto users and choices of the users are monitored. This can be done byusing, as one example, a crowd sourcing website or algorithm whereindividuals are paid a small amount for providing an opinion. Targetimage T1, and a set of candidate images, C1, C2, C3 to be compared totarget image T1 are provided to judges. Each candidate image has a rangeassociated with it, such as a scale of 1 to 5, going sequentially fromlow to high, for example. Each judge is asked how similar a candidate,such as C2, is to the target image T1, Hundreds of thousands ofjudgments can be taken, every target image to every candidate image. Theecommerce system knows what categories the target images are in, andtheir color, shape and, at least potentially, texture. For example, theecommerce system knows how similar the color of T1 is to, say, C1, andthe color of T1 is to, say, C3, and the color of T1 to, say, C3. Thesame is known as to shape and the same is known as to texture, and thelike. From the foregoing information, the system can calculate, for agiven category, say C2, all of the target images, and the candidatecomparisons and determine, for example, that the feature of thecandidate that was most highly correlated with judgment scores was, forexample, color. So for category C2, color would be weighted, as w,higher than, say, shape or texture. To determine the optimum combinationof weights across all features, an approach such as linear regressioncould be used, and many other modeling techniques could also be applied.This process can be applied with item attributes like brand, material,pattern, style, size, dress length, and sleeve type, and others. Theresult would then be obtaining judgments that, say, for shoes peoplemight care more about brand, whereas for dresses, people might care moreabout pattern, as just one example. Alternatively, the system can countclick-through patterns form the ecommerce website to determine userweighting preferences of attributes for given categories.

More Like This

An example operation of the image similarity search function describedabove can be seen from FIGS. 16, 17, 18, and 19. FIG. 16 shows a homepage of an ecommerce website. Using a left navigation pane, one cannavigate to “Fashion” button 161, “Women's Handbags” button 163, andarrive at the page seen in FIG. 17 by way of a first pass similaritysearch using a TF-IDF-like algorithm as discussed previously, and then asecond-pass ranking of for example, the top N results selected in thefirst pass similarity search to place a resulting number of similaritems on the page illustrated in FIG. 17. The second pass similaritysearch can be done by any of a number of operators as discussed above.Each second pass similarity search yields the grouping of handbags thatmeets the type of second pass similarity search used to rank the top Nitems in the first pass similarity search. In addition, certain secondpass similarity searches can be user-selected such as in FIG. 17, inwhich the second pass similarity search is by time: sale ending soonestas seen at “Sort By” box 171. “Sort By” box 171 can be a drop-down menufor some of the second pass similarity searches discussed above (e.g.,by newly listed items; by price+shipping cost (lowest first); byprice+shipping cost (highest first); by best match; by price (highestfirst); by price (lowest first)), which can be user-selected by clickingon choices for the foregoing that are selected through the drop downmenu at box 171.

Continuing with the discussion of FIG. 17, if one hovers her mouse overany bag, such as the bag shown in image 173, a “More Like This” choicewill be viewable. Selecting “More Like This” will yield a similaritysearch on the image 173 such as that shown in FIG. 18 where handbagssimilar to the one shown in image 173 appear. These bags will bedetermined by the above-explained edge detection, color detection,and/or texture detection, applied to the query search from the “MoreLike This” similarity search as discussed in detail above with referenceto FIG. 17. One can continue similarity searching “More Like This” toadditional depths, for example, by hovering one's mouse over image 181of FIG. 18 and clicking again on “More Like This” for the handbag shownin the image 181, yielding yet another page of handbags (not shown),these similar to the one shown in the image 181. A test for an endsignal can be made before searching to a next additional depth. If anend signal is detected, such as a user accessing an application notsupported by the search, or such as the user otherwise discontinuing thesearch, among other end signals, the method ends. If an end signal isnot detected at the test, the searching continues to at least oneadditional depth, A “bread crumb” trail can be kept so that the user cantrack her similarity searches and go back, if desired, to the handbagshe likes best of the handbags discovered through various depths of“More Like This.”

If desired, the “More Like This” similarity search can be designed sothat one can enter the “More Like This” similarity search by way of asimilarity search at “All Categories” button 165, for example, of FIG.16, initially, and proceed to the desired category of the group ofcategories that can appear from selecting the “All Categories” button165, and following the above described similarity search of “More LikeThis” depths. Alternatively, one can design the “More Like This”similarity search such that one can enter the similarity search by aword search, for example, by entering “Coach Leather Handbag” at theword search box 167, for example, at FIG. 16, and continue as discussedabove.

The system can be designed such that at any level of the above “MoreLike This” similarity searches, regardless of the method of entering thesearch, the system can allow the user to switch from one category ofitem to another category of item. For example, when in the “More LikeThis” similarity search on FIG. 18, discussed above, instead of clickingon “More Like This” for the handbag shown in image 181, one can changethe item category and enter a new similarity search for, say, women'sshoes, using any of the entry points that were discussed above for thecase of women's handbags, thereby beginning a new “More Like This”similarity search for the case of women's shoes, searching to any level,again subject to an end signal as discussed above. Again, a bread crumbtracking trail can be kept of the sequence of “More Like This” levels ofsimilarity searches for women's handbags and the sequence of “More LikeThis” levels of similarity searches for women's shoes, so that the usercan at any point, using the bread crumb tracking trail, go back andpurchase any of the located women's handbags or women's shoes.Experience has shown that keeping prospective buyers in the similaritysearch with additional “More Like This” similarity searches to variouslevels and for items of various categories has been seen to lead to moresales.

FIG. 19 is an illustration of the “More Like This” similarity searchdescribed above. The user may enter a search query using a textualsearch as discussed above at operation 190. This may be a search entryby way of the “Fashion” button 161 or may be by way of a word similaritysearch by entering a search term into the search box 175 of FIG. 17. Asdiscussed above, “Textual” means text and image information to be storedabout the item the seller is listing, including item title, andattributes of the item. This can be considered “structured” and“unstructured” item data. For example, unstructured data may includeitem title and item description, which may be data in which the selleris free to include whatever the seller wants to include. Structured datamay be data such as the item category (e.g., shoes, handbags, and thelike) and item aspects such as brand, material, and the like, where eachpiece of information can be stored as separate name-value pairs. So, ifthe listing item is a Coach handbag with title “New brown Coach purse”,the digest may include the following terms as textual elements of apre-computed digest (new, brown, Coach, purse, Brand: Coach, Material:leather, Style: satchel).

Continuing with the method, a textual “More Like This” search may betriggered through operation 190 of FIG. 19, which could be entered byway of, for example, the button 161 of FIG. 16. Alternatively, thesearch may be a word search (e.g. “pink leather brooch”) entered atoperation 191 of FIG. 19, which may be entered through, for example, thesearch box 167 of FIG. 16. In either case, the system detects the entryof the search and detect the item category of the search at operation193, for example, by detecting that “Women's Handbags” is selected atbutton 163 in FIG. 16. Next, the system performs a first pass similaritysearch at operation 194. The system will then perform a second passsimilarity search on the top N items found by the first pass similaritysearch. Again, the second pass similarity search at operation 195 can beperformed using a best match algorithm or any of the operators discussedpreviously, some or all of which can be user-selectable as by way of,for example, a drop-down menu at the box 171 of FIG. 17. When the secondpass similarity is completed, the system can provide one or more signalsfor rendering the search results at operation 196.

At this point, the system can test for an end signal and if an endsignal is detected, the method ends. If no end signal is detected, thesystem continues to operation 197 to test for a change of item category.If the system detects a change of item category, such as the userentering a new category by any of the buttons 161, 165, or box 167 ofFIG. 16, the system returns to operation 193 to detect the new itemcategory and continues as discussed above. If, on the other hand, theuser does not change item category, the system detects at decisionoperation 198 whether the user is entering a “More Like This” similaritysearch as at image 181 of FIG. 18. If the answer at operation 198 is no,the search ends at operation 199. If the answer is yes, the systemreturns to perform the first pass similarity search at operation 194 andcontinues with the rest of the search as discussed above.

It will further be appreciated that the system and methodology describedabove can be applied to video content as well as or instead of imagedata. The method may in such case include extracting images from videofiles, and processing the extracted images in accordance with theexample embodiments described herein. Image extraction from such videocontent may include automatically identifying scene changes in the videocontent (e.g., by comparison of successive frames or images in the videocontent), and extracting images or frames at the start of such scenechanges.

The example embodiments described herein address some of the technicalchallenges associated with effective processing of images linked to itemlistings. For instance, image databases of network-based marketplacesare often very large, comprising millions of images, so that the timeand/or processing power consumed by conducting a search of imagecomparison in the database is prohibitive. However, a comparison ofindex data generated for the query image with index data of databaseimages is considerably less resource intensive,

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. A component is a tangible unitcapable of performing certain operations and may be configured orarranged in a certain manner. In example embodiments, one or morecomputer systems (e.g., a standalone, client or server computer system)or one or more components of a computer system (e.g., a processor or agroup of processors) may be configured by software (e.g., an applicationor application portion) as a component that operates to perform certainoperations as described herein.

In various embodiments, a component may be implemented mechanically orelectronically. For example, a component may comprise dedicatedcircuitry or logic that is permanently configured (e.g., as aspecial-purpose processor) to perform certain operations. A componentmay also comprise programmable logic or circuitry (e.g., as encompassedwithin a general-purpose processor or other programmable processor) thatis temporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a componentmechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) maybedriven by cost and time considerations.

Accordingly, the term “component” should be understood to encompass atangible entity, be that an entity that is physically constructed,permanently configured (e.g., hardwired) or temporarily configured(e.g., programmed) to operate in a certain manner and/or to performcertain operations described herein. Considering embodiments in whichcomponents are temporarily configured (e.g., programmed), each of thecomponents need not be configured or instantiated at any one instance intime. For example, where the components comprise a general-purposeprocessor configured using software, the general-purpose processor maybe configured as respective different components at different times.Software may accordingly configure a processor, for example, toconstitute a particular component at one instance of time and toconstitute a different component at a different instance of time.

Components can provide information to, and receive information from,other components. Accordingly, the described components may be regardedas being communicatively coupled. Where multiple of such componentsexist contemporaneously, communications may be achieved through signaltransmission (e.g., over appropriate circuits and buses) that connectthe components. In embodiments in which multiple components areconfigured or instantiated at different times, communications betweensuch components may be achieved, for example, through the storage andretrieval of information in memory structures to which the multiplecomponents have access. For example, one component may perform anoperation, and store the output of that operation in a memory device towhich it is communicatively coupled. A further component may then, at alater time, access the memory device to retrieve and process the storedoutput. Components may also initiate communications with input or outputdevices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of some of the operationsmay be distributed among the one or more processors, not only residingwithin a single machine, but deployed across a number of machines. Insome example embodiments, the processor or processors may be located ina single location (e.g., within a home environment, an officeenvironment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., Application Program Interfaces APIs).)

Electronic Apparatus and System

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations thereof.Example embodiments may be implemented using a computer program product,e.g., a computer program tangibly embodied in an information carrier,e.g., in a machine-readable medium for execution by, or to control theoperation of, data processing apparatus, e.g., a programmable processor,a computer, or multiple computers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry,e.g., a field programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Inembodiments deploying a programmable computing system, it will beappreciated that that both hardware and software architectures requireconsideration. Specifically, it will be appreciated that the choice ofwhether to implement certain functionality in permanently configuredhardware (e.g., an ASIC), in temporarily configured hardware (e.g., acombination of software and a programmable processor), or a combinationof permanently and temporarily configured hardware may be a designchoice. Below are set out hardware (e.g., machine) and softwarearchitectures that may be deployed, in various example embodiments.

Example Three-Tier Software Architecture

In some embodiments, the described methods may be implemented using oneof a distributed or non-distributed software application designed undera three-tier architecture paradigm. Under this paradigm, various partsof computer code (or software) that instantiate or configure componentsor modules may be categorized as belonging to one or more of these threetiers. Some embodiments may include a first tier as an interface (e.g.,an interface tier). Further, a second tier may be a logic (orapplication) tier that performs application processing of data inputtedthrough the interface level. The logic tier may communicate the resultsof such, processing to the interface tier, and/or to a backend, orstorage tier. The processing performed by the logic tier may relate tocertain rules, or processes that govern the software as a whole. Athird, storage tier, may be a persistent storage medium, or anon-persistent storage medium. In some cases, one or more of these tiersmay be collapsed into another, resulting in a two-tier architecture, oreven a one-tier architecture. For example, the interface and logic tiersmay be consolidated, or the logic and storage tiers may be consolidated,as in the case of a software application with an embedded database. Thethree-tier architecture may be implemented using one technology, or, avariety of technologies. The example three-tier architecture, and thetechnologies through which it is implemented, may be realized on one ormore computer systems operating, for example, as a standalone system, ororganized in a server-client, peer-to-peer, distributed or some othersuitable configuration. Further, these three tiers may be distributedbetween more than one computer systems as various components.

Components

Example embodiments may include the above described tiers, and processesor operations about constituting these tiers may be implemented ascomponents. Common to many of these components is the ability togenerate, use, and manipulate data. The components, and thefunctionality associated with each, may form part of standalone, client,server, or peer computer systems. The various components may beimplemented by a computer system on an as-needed basis. These componentsmay include software written in an object-oriented computer languagesuch that a component oriented, or object-oriented programming techniquecan be implemented using a Visual Component Library (VCL), ComponentLibrary for Cross Platform (CLX), Java Beans (JB), Java Enterprise Beans(EJB), Component Object Model (COM), Distributed Component Object Model(DCOM), or other suitable technique.

Software for these components may further enable communicative couplingto other components (e.g., via various Application Programminginterfaces (APIs)), and may be compiled into one complete server,client, and/or peer software application. Further, these APIs may beable to communicate through various distributed programming protocols asdistributed computing components.

Distributed Computing Components and Protocols

Some example embodiments may include remote procedure calls being usedto implement one or more of the above described components across adistributed programming environment as distributed computing components.For example, an interface component (e.g., an interface tier) may formpart of a first computer system that is remotely located from a secondcomputer system containing a logic component (e.g., a logic tier). Thesefirst and second computer systems maybe configured in a standalone,server-client, peer-to-peer, or some other suitable configuration.Software for the components may be written using the above describedobject-oriented programming techniques, and can be written in the sameprogramming language, or a different programming language. Variousprotocols may be implemented to enable these various components tocommunicate regardless of the programming language used to write thesecomponents. For example, a component written in C++ may be able tocommunicate with another component written in the Java programminglanguage through utilizing a distributed computing protocol such as aCommon Object Request Broker Architecture (CORBA), a Simple ObjectAccess Protocol (SOAP), or some other suitable protocol. Someembodiments may include the use of one or more of these protocols withthe various protocols outlined in the Open Systems Interconnection (OSI)model, or Transmission Control Protocol/Internet Protocol (TCP/IP)protocol stack model for defining the protocols used by a network totransmit data.

A System of Transmission Between a Server and Client

Example embodiments may use the OSI model or TCP/IP protocol stack modelfor defining the protocols used by a network to transmit data. Inapplying these models, a system of data transmission between a serverand client, or between peer computer systems may for example includefive layers comprising: an application layer, a transport layer, anetwork layer, a data link layer, and a physical layer. In the case ofsoftware, for instantiating or configuring components, having athree-tier architecture, the various tiers (e.g., the interface, logic,and storage tiers) reside on the application layer of the TCP/IPprotocol stack. In an example implementation using the TCP/IP protocolstack model, data from an application residing at the application layeris loaded into the data load field of a TCP segment residing at thetransport layer. This TCP segment also contains port information for arecipient software application residing remotely. This TCP segment isloaded into the data load field of an IP datagram residing at thenetwork layer. Next, this IP datagram is loaded into a frame residing atthe data link layer. This frame is then encoded at the physical layer,and the data transmitted over a network such as an internet, Local AreaNetwork (LAN), Wide Area Network (WAN), or some other suitable network.In some cases, internet refers to a network of networks. These networksmay use a variety of protocols for the exchange of data, including theaforementioned TCP/IP, and additionally ATM, SNA, SDI, or some othersuitable protocol. These networks may be organized within a variety oftopologies (e.g., a star topology), or structures.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the embodiment. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Data Structures

FIG. 21 is a high-level entity-relationship diagram of an exampleembodiment, illustrating various tables 700 that may be maintainedwithin the databases 35 to 37, and that are utilized by and support theapplications 30 and 32. A user table 702 contains a record for eachregistered user of the networked system 12, and may include identifier,address and financial instrument information pertaining to each suchregistered user. A user may operate as a seller, a buyer, or both,within the networked system 12. In one example embodiment, a buyer maybe a user that has accumulated value (e.g., commercial or proprietarycurrency), and is accordingly able to exchange the accumulated value foritems that are offered for sale by the networked system 12.

The tables 700 also include an items table 704 in which are maintaineditem records for goods and services that are available to be, or havebeen, transacted via the networked system 12. Each item record withinthe items table 704 may furthermore be linked to one or more userrecords within the user table 702, so as to associate a seller and oneor more actual or potential buyers with each item record.

The items table 704 may be connected to an image table which containsimages associated with the respective items or item listings in theitems table 704. The image table 720 is in turn connected to an indexdata table 730 which contains index data as described in detail above.

A transaction table 706 contains a record for each transaction (e.g., apurchase or sale transaction) pertaining to items for which recordsexist within the items table 704.

An order table 708 is populated with order records, each order recordbeing associated with an order. Each order, in turn, may be with respectto one or more transactions for which records exist within thetransaction table 706.

Bid records within a bids table 710 each relate to a bid received at thenetworked system 12 in connection with an auction-format listingsupported by an auction application 32. A feedback table 712 is utilizedby one or more reputation applications 50, in one example embodiment, toconstruct and maintain reputation information concerning users. Ahistory table 714 maintains a history of transactions to which a userhas been a party. One or more attributes tables 716 record attributeinformation pertaining to items for which records exist within the itemstable 704. Considering only a single example of such an attribute, theattributes tables 716 may indicate a currency attribute associated witha particular item, the currency attribute identifying the currency of aprice for the relevant item as specified in by a seller.

FIG. 20 shows a diagrammatic representation of a machine in the exampleform of a computer system 500 within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed. In alternative embodiments, themachine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be aserver computer, a client computer, a personal computer (PC), a tabletPC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, a web appliance, a network router, switch or bridge, or anymachine capable of executing a set of instructions sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 500 includes a processor 502 (e.g., acentral processing unit (CPU) a graphics processing unit (GPU) or both),a main memory 504 and a static memory 506, which communicate with eachother via a bus 508. The computer system 500 may further include a videodisplay unit 510 (e.g., a liquid crystal display (LCD) or a cathode raytube (CRT)). The computer system 500 also includes an alphanumeric inputdevice 512 (e.g., a keyboard), a cursor control device 514 (e.g., amouse), a disk drive unit 516, a signal generation device 518 (e.g., aspeaker) and a network interface device 520.

The disk drive unit 516 includes a machine-readable medium 522 on whichis stored one or more sets of instructions (e.g., software 524)embodying any one or more of the methodologies or functions describedherein. The software 524 may also reside, completely or at leastpartially, within the main memory 504 and/or within the processor 502during execution thereof by the computer system 500, the main memory 504and the processor 502 also constituting machine-readable media.

The software 524 may further be transmitted or received over a network526 via the network interface device 520.

While the machine-readable medium 522 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies described herein. The term “machine-readable medium” shallaccordingly be taken to include, but not be limited to, solid-statememories, optical and magnetic media, and carrier wave signals.

Thus, a method and system to index images and to perform an image-basedsearch in a network-based marketplace have been described. Although thepresent method and system have been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the application. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

What is claimed is:
 1. A computer implemented method comprising:receiving a query comprising a query image that is sent via a userinterface at a client machine operating over a wireless communicationchannel; responsive to receiving the query image, searching a databaseand obtaining from the database a set of images that are similar to thequery image, the searching a database and obtaining a set of images thatare similar to the query image comprising a two-pass search, a firstpass resulting in a plurality of images that are similar to the queryimage and ranking the plurality of images that are similar to the queryimage to form a similarity score, and a second pass comprising rankingfewer than all of the resulting plurality of images from the first passagainst pre-computed digests of image-based listings comprising textualinformation, wherein the ranking of fewer than all of the resultingplurality of images is performed using a best match procedure thatcomprises weighting the resulting plurality of images by colorsimilarity, by edge similarity, and by the similarity score; providing,over the wireless communication channel for display at the userinterface at the client machine, results of the searching, the resultscomprising image members of the obtained set of images that are similarto the query image; detecting a request, sent via the user interface atthe client machine, for a similarity search for more images similar toone of the image members of the set of images displayed at the userinterface at the client machine; responsive to detecting the requestsent via the user interface at the client machine for the similaritysearch, searching for more images similar to one of the image members ofthe set of images displayed at the user interface at the client machine,searching the database and obtaining from the database the more imagessimilar to the one of the image members of the set of images; andresponsive to the searching the database and the obtaining from thedatabase more images similar to the one of the image members of the setof images, providing, over the wireless communication channel fordisplay at the user interface at the client machine, at least some ofthe obtained more images similar to the one of the image members of theset of images.
 2. The method of claim 1, wherein weighting the resultingplurality of images by color similarity comprises performing the secondpass by ranking fewer than all of the resulting plurality of imagesbased on a distance between color histograms of the resulting pluralityof images that have color attributes.
 3. The method of claim 1, whereinweighting the resulting plurality of images by edge similarity comprisesperforming the second pass by ranking fewer than all of the resultingplurality of images based on cosine similarity of edge matrices of onesof the fewer than all of the resulting plurality of images that haveedge attributes that are computed using edge matrices.
 4. The method ofclaim 1, wherein the two-pass search comprises performing the secondpass by ranking fewer than all of the resulting plurality of imagesusing a pattern comparison of ones of the fewer than all of theresulting plurality of images that have pattern attributes.
 5. Themethod of claim 1, wherein using the two-pass search comprisesperforming the second pass by ranking fewer than all of the resultingplurality of images based on text similarity of ones of the fewer thanall of the resulting plurality of images that have text attributes. 6.The method of claim 1, wherein at least one of the image memberscomprises sale attributes, the method further comprising ranking fewerthan all of the image members by at least one of the sale attributes. 7.The method of claim 6, wherein the sale attributes comprise one of time(sale ending soonest); newly listed; price+shipping cost (lowest first);price+shipping cost (highest first); price (highest first); price(lowest first), seller or item quality.
 8. The method of claim 1,further comprising determining whether an end signal is received; andbased on the end signal not being received, searching the database andobtaining from the database a set of images similar to a subsequentquery image received via the user interface at the client machine. 9.The method of claim 8 wherein the query image comprises an image of aproduct of a first product category and the subsequent query imagecomprises an image of a product of a second product category.
 10. Themethod of claim 1 wherein the query comprises structured data andunstructured data.
 11. The method of claim 1 wherein the searching thedatabase and obtaining from the database a set of images that aresimilar to the query image comprises searching in a first pass and asecond pass, wherein the second pass is selectable via the userinterface at the client machine.
 12. The method of claim 11 wherein thefirst pass is made using a TF-IDF algorithm.
 13. The method of claim 1,the second pass of the two pass search comprising ranking images thatresult from the first pass of the two-pass search against pre-computeddigests of image-based listings comprising textual information, theranking performed using feedback from users.
 14. The method of claim 1wherein the displayed results comprising image members of the obtainedset of images similar to the query image comprise selectable icons that,when selected, comprise the request by the user at the client machinefor a similarity search for more images similar to one of the imagemembers of the set of images.
 15. A computer-readable hardware storagedevice having embedded therein a set of instructions which, whenexecuted by one or more processors of a computer, causes the computer toexecute operations comprising: receiving a query comprising a queryimage that is sent via a user interface at a client machine operatingover a wireless communication channel; responsive to receiving the queryimage, searching a database and obtaining from the database a set ofimages that are similar to the query image, the searching a database andobtaining a set of images that are similar to the query image comprisinga two-pass search, a first pass resulting in a plurality of images thatare similar to the query image and ranking the plurality of images thatare similar to the query image to form a similarity score, and a secondpass comprising ranking fewer than all of the resulting plurality ofimages from the first pass against pre-computed digests of image-basedlistings comprising textual information, wherein the ranking of fewerthan all of the resulting plurality of images is performed using a bestmatch procedure that comprises weighting the resulting plurality ofimages by color similarity, by edge similarity, and by the similarityscore; providing, over the wireless communication channel, for displayat the user interface at the client machine, results of the searching,the results comprising image members of the obtained set of images thatare similar to the query image; detecting a request via the userinterface at the client machine for a similarity search for more imagessimilar to one of the image members of the set of images displayed atthe user interface at the client machine; responsive to detecting therequest via the user interface at the client machine for the similaritysearching for more images similar to one of the image members of the setof images displayed at the user interface at the client machine,searching the database and obtaining from the database more imagessimilar to the one of the image members of the set of images; andresponsive to the searching the database and obtaining from the databasemore images similar to the one of the image members of the set ofimages, providing, over the wireless communication channel for displayat the user interface at the client machine, at least some of theobtained more images similar to the one of the image members of the setof images.
 16. The computer-readable hardware storage device of claim15, wherein weighting the resulting plurality of images by edgesimilarity comprises performing the second pass by ranking fewer thanall of the resulting plurality of images based on cosine similarity ofedge matrices of ones of the fewer than all of the resulting pluralityof images that have edge attributes that are computed using edgematrices.
 17. The computer-readable hardware storage device of claim 15,wherein weighting the resulting plurality of images by edge similaritycomprises performing the second pass by ranking fewer than all of theresulting images based on cosine similarity of edge matrices of ones ofthe fewer than all of the resulting images that have edge attributescomputed using edge matrices.
 18. The computer-readable hardware storagedevice of claim 15, wherein the two-pass search comprises performing thesecond pass by ranking fewer than all of the resulting plurality ofimages using a pattern comparison of ones of the fewer than all of theresulting plurality of images that have pattern attributes.
 19. Thecomputer-readable hardware storage device of claim 15, wherein using thetwo-pass search comprises performing the second pass by ranking fewerthan all of the resulting plurality of images based on text similarityof ones of the fewer than all of the resulting plurality of images thathave text attributes.
 20. The computer-readable hardware storage deviceof claim 15, wherein at least one of the image members comprises saleattributes.
 21. The computer-readable hardware storage device of claim20, wherein the sale attributes comprise one of time (sale endingsoonest); newly listed; price+shipping cost (lowest first);price+shipping cost (highest first); price (highest first); price(lowest first), seller or item quality.
 22. The computer-readablehardware storage device of claim 15, wherein at least one of the imagemembers, when rendered at the user interface of the client machine, isselectable as a subsequent query image.
 23. The computer-readablehardware storage device of claim 15, the operations further comprisingdetermining if an end signal is received; and based on the end signalnot being received, performing a further search based on a subsequentquery having a query image received via the user interface at the clientmachine.
 24. The computer-readable hardware storage device of claim 23wherein the query image comprises an image of a product of a firstproduct category and the subsequent query image comprises an image of aproduct of a second product category.
 25. The computer-readable hardwarestorage device of claim 15 wherein the query comprises structured dataand unstructured data.
 26. The computer-readable hardware storage deviceof claim 15 wherein the first pass is made using a TF-IDF algorithm. 27.A system comprising: at least one computer processor and storageconfigured to receive a query comprising a query image that is sent viaa user interface at a client machine operating over a wirelesscommunication channel, and search a database and obtain from thedatabase a set of images similar to the query image, the search of adatabase and obtainment of a set of images that are similar to the queryimage comprising a two-pass search, a first pass resulting in aplurality of images that are similar to the query image and ranking theplurality of images that are similar to the query image to form asimilarity score, and a second pass comprising ranking fewer than all ofthe resulting plurality of images from the first pass againstpre-computed digests of image-based listings comprising textualinformation, wherein the ranking of fewer than all of the resultingplurality of images is performed using a best match procedure thatcomprises weighting the resulting plurality of images by colorsimilarity, by edge similarity, and by the similarity score; and atransmission circuit configured to transmit, for display at the userinterface at the client machine, the results of the search of thedatabase, the results comprising image members of the set of images thatare similar to the query image, wherein the at least one computerprocessor and storage is further configured to detect a request via theuser interface at the client machine for a similarity search for moreimages similar to one of the image members of the set of imagesdisplayed at the user interface at the client machine, and responsive todetection of the request for the similarity search for images similar toone of the image members of the set of images displayed at the userinterface at the client machine, search the database and obtain from thedatabase more images similar to the one of the image members of the setof images, and responsive to the search of the database, and theobtained more images similar to the one of the image members of the setof images, provide, for display at the user interface at the clientmachine, at least some of the obtained more images similar to the one ofthe image members of the set of images.